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What is disclosed is a method for removing noise by 
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to color interpolation, and uses techniques suited to the 
classification, whether edge or non-edge. 

22 Claims, 7 Drawing Sheets 



G i0 G iE G i H 

R G2 R G 2 R G 2 

G 10 G l@ G l[jLl =^> BLUE SUB-IMAGE 
R Gj R G 2 R G 2 

R G 2 R G 2 R G 2 



1 2 3 4 5 6 

\2H B ED B ED B 

R G 2 R G2 R G2 

R G 2 R G2 R G 2 
[2llB |57] B [Gj] B 
R G 2 R R Gj 

Q 1 SUB-IMAGE 



1 2 3 



5 6 



Gj B G| B Gj 

R^R [G^] R 

Gj B Gj B G { 

rJc^Jr [g^r 

G] B G] B G] 

R[C§]R jGjjR [C 

G 2 SUB- IMAGE 



B 



B 



B 



3 4 



Gj B Gj B Gj B 
002 002 002 

Gj B G[ B Gj B 

Qg 2 0g 2 [r]g 2 =^RED SUB-IMAGE 
Gj B G] B G[ B 
002002 002 



11/04/2003, EAST Version: 1.4.1 



U.S. Patent May 8, 2001 Sheet 1 of 7 US 6,229,578 Bl 



110 



nPPIMF imitiai i nrAi i7A-nmi 

regFon 



I 



120 



DETERMINE GRADIENT OF EVERY 
PIXEL USING OPERATOR OR MASK 
IN LOCALIZATION REGION 




150 



APPLY LINEAR 
AVERAGE FILTERING 
ON NEIGHBORING 
PIXELS OF SIMILAR 
INTENSITY 



APPLY MULTI-LEVEL MEDIAN 
HYBRID FILTERING ON PIXEL 




175 



FETCH NEXT 

PIXEL'S 
INFORMATION 



180 



DEFINE NEW LOCALIZATION 
REGION 



Fig.i 



11/04/2003, EAST Version: 1.4.1 



U.S. Patent May 8, 2001 Sheet 2 of 7 US 6,229,578 Bl 



IMAGE I 






h 


h 




k 


15 


k 


17 


h 





0 


A4 


A 2 


0 




A 3 


0 


A6 



MASK A 



AX= A { Iq + A 2 I 3 + A3 Ig 
+ A4 I 2 + A5 I 5 + Ag I g 



Fig. 2a 



IMAGE 






11 


12 


13 




15 


16 


17 





Bl 


»2 


B3 


0 


0 


0 


B 4 


B5 


B6 



MASKS. 



AY=Bj Iq +B 2 Ii +B 3 I3 
+ B 4 I 6 +B 5 I 7 +B 6 Ig 



Fig. 2b 



11/04/2003, EAST Version: 1.4.1 



U.S. Patent May 8, 2001 Sheet 3 of 7 



US 6,229,578 Bl 



I 



1 
2 
3 
4 
5 
6 



1 2 



Gj B Gj B Gj B 
R G 2 R G 2 R G 2 
G 10 G 10 G 1 



B 



R G 2 R G 2 R G 2 

G i0 G i [jlI g i Dl 

R G 2 R G 2 R G 2 



BLUE SUB-IMAGE 



1 
2 
3 
4 
5 
6 



1 2 



Gi B Gj B Gj 



B 



R G 2 R G 2 R G 2 
B 



Gj B Gj B Gj 



B 



R G 2 R G 2 R G 2 
B 



Gl 



B |GjJ B 
R G 2 R G 2 R G 2 

G 1 SUB-IMAGE 



1 



4 5 



Gj B Gj B Gi 
R 



B 



G 2 R G 2 R G 2 



Gj B G t B Gj 
R 



B 



G 2 R G 2 R G 2 



Gj B Gi B Gi 
R 



B 



G 2 



R |G2j R 
G 2 SUB-IMAGE 



G 2 



Gj B Gj B Gi 
G 2 



R 



R 



Gj B Gj B Gj 
G 2 



R 



R 



Gj B Gj B Gi 

~r]g 2 [r~ 



G 2 
B 
G 2 
B 

G 2 



R 



R 



R 



B 

G 2 

B 

G 2 

B 

G 2 



RED SUB-IMAGE 



Fig. 3 



11/04/2003, EAST Version: 1.4.1 



U.S. Patent May 8, 2001 Sheet 4 of 7 



US 6,229,578 Bl 



/ 





1 


2 


3 , 


f 4 


5 




1 


NE 


NE 


/ 


NE 




NE 


2 


NE 


/ 


NE 


NE 


I 


NE 


3 


NE 


— (— 
i. 


NE 


NE 


— \- 
H\ 


NE 


4 


NE 




N E 1 


NE 


NE 


E*\ 
1 


5 


NE 


NE 




NE 


NE 


%! 


6 








NE 


?3_ 


1 — r 

E 3 I 
_ > 


E 


f y \ 
- EDGE PIXELS 









• E x - FIRST EDGE FEATURE, 

• E 2 - SECOND EDGE FEATURE, 

• E 3 - THIRD EDGE FEATURE, 

• E 4 - FOURTH EDGE FEATURE 
NE - NON-EDGE PIXELS 



Fig. 4 



11/04/2003, EAST Version: 1.4.1 



U.S. Patent May 8, 2001 Sheet 5 of 7 US 6,229,578 Bl 



Yeast(U)- 

x(i.j) - 
YwEST('j)- 

Y NORTH('j)- 
X(ij)' 

Ysouth('J)- 



Y NORTHEASTti.J> 
x(i,|) - 

Y SOUTHWEST('.i)- 

y northwest(m)- 

x(i,j)" 

Y SOUTHEAST(i.j)- 



MEDIAN 
FILTER 



510 
V 

Mi 



MEDIAN 
FILTER 



512 
M 2 



MEDIAN 
FILTER 



514 
V 

M 3 



MEDIAN 
FILTER 



516 
V 

M 4 



x(ij)- 



MEDIAN 
FILTER 



520 

J 

M 5 



MEDIAN 
FILTER 



522 

J 



MEDIAN 
FILTER 



530 

J 



— x(i,j) 



M 6 



Fig. 5 



11/04/2003, EAST Version: 1.4.1 



U.S. Patent May 8, 2001 Sheet 6 of 7 US 6,229,578 Bl 





f 


1 ' ■"- 


FIND ABSOLUTE VALUE DIFFERENCES 
(AVDs) BETWEEN ADJACENT PIXELS 
AND EDGE PIXEL x(i,j) 60 5 






FIND AVERAGE OF AVDs 61Q 






FETCH FIRST ADJACENT PIXEL 




EXCLUDE ADJACENT PIXEL 



FROM AVERAGING SEJ^ 



FETCH NEXT ADJACENT 
PIXEL 

645 



INCLUDE ADJACENT PIXEL 
IN AVERAGING SET 62C 



ADD ADJACENT PIXEL'S 
VALUE TO ACCUM 62J 



INCREMENT 
NUMCOUNT 630 




ADD x(i,j) TO ACCUM 



65J 



INCREMENT 
NUMCOUNT 665 



DIVIDE ACCUM BY 
NUMCOUNT 670 



STORE RESULT AS NEW PIXEL 
VALUE 675 



PROCESS NEXT EDGE PIXEL 
685 



CLEAR ACCUM AND 
NUMCOUNT 680 



Fig. 6 



11/04/2003, EAST version: 1.4.1 



U.S. Patent May 8, 2001 Sheet 7 of 7 US 6,229,578 Bl 



COMPUTER SYSTEM 



PROCESSOR 



7 ^ 



716 



MONITOR 720 



_712 



MEMORY 



DISK 



_711 



BRIDGE 714 



-SYSTEM BUS 
713 



DISPLAY 
ADAPTER 



I/O BUS 
715 



I/O PORT hJ 17 




RENDERED 

IMAGE 

750 



J 



Fig. 7 




IMAGE 
PROCESSING 
CIRCUIT 
732 



IMAGE 

MEMORY 

UNIT 

734 OBJECT/SCENE 
740 




^CAMERA 
730 



11/04/2003, EAST Version: 1.4.1 



US 6,229 : 

1 

EDGE-DETECTION BASED NOISE 
REMOVAL ALGORITHM 

BACKGROUND OF THE INVENTION 

1. Meld ot the Invention 5 
The invention relates generally to the field of image 

processing. More specifically, the invention relates process- 
ing of images generated by a digital imaging device. 

2. Description of the Related Art 

In the art of image processing, raw images of an object/ 
scene captured from a sensing or capture device are often 
subject to varying types of "noise" (elements not present in 
the object or environment which may nonetheless appear in 
the image). The noise present in an image may be due to the 15 
characteristics of the imaging system such as the sensor or 
processing steps subsequent to the initial image capture 
which may add noise while trying to achieve a different 
purpose. The properties and characteristics that would indi- 
cate that a pixel or region of pixels is "noisy" and the 2 q 
properties that would indicate a pixel or region of pixels is 
an edge or a fine detail of the image are difficult to 
distinguish. Thus, a fundamental problem with the removal 
of noise is that often a removal of what is indicated as noise 
may actually be a removal of fine edge or detail. If the fine 25 
detail or edge is removed, a blurring effect may occur within 
that region of the image further, in color images, the blurring 
effect leads to a bleeding of one color across the edge to 
another pixel(s). Noise removal procedures that were based 
upon linear filtering techniques suffered greatly from this 30 
malady and thus, a class of filtering techniques based on 
ranked order statistics such as the median were developed. 

The median filter ranks in order the intensity values 
belonging to a pixel P (for which the filter is being applied) 
and pixels in a particular neighborhood or along a particular 35 
vector about a pixel R For example, a median filter (applied 
in a particular direction(s) through the pixel to neighboring 
pixels) applied to sample values including and about the 
pixel P of {12, 13, 200, 50, 14} would first be ranked in 
order as {12, 13, 14, 118, 200}. The so-called uni-directional 40 
FIR median hybrid filter would replace the original pixel 
location P that had a value of 200 with the median of the 
sample set which is 14. Thus, the output vector, after the 
filter, would be: {12, 13, 14, 50, 14}. If the value 200 were 
in fact part of an edge rather than noise, the smoothing 45 
caused by applying the filter as shown in the output vector 
values would decimate the edge feature. 

Several improved median filters have been developed to 
compensate for this problem. One particular such median 
filter, the multilevel FIR (Finite Impulse Response) median 50 
hybrid filter repeatedly takes the median filter in each 
direction about an image and applies at each filter the 
original input pixel. The muld- level median hybrid filter has 
averaging sub-filters that reduce the burden of sorting opera- 
tions by averaging pixels in a particular filter direction, and 55 
then performing the median computation upon a smaller set 
of values, such as three. Thus, in a median hybrid filter, two 
neighboring west pixels would be averaged and the result 
fed to a median filter along with the average of two neigh- 
boring east pixels. The third input to the median filter is the 60 
pixel under consideration for noise removal. In other 
directions, a similar procedure is applied. In a three-level 
median hybrid filter, the first level pairs all such averaged 
neighboring pixels with vectors in opposing directions 
(north with south, etc.) and for each pair of direction 65 
averages (8 of them) feeds these into a median filter also 
along with the pixel of concern as a third input. The resulting 
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median values of the first filter are again paired and along 
with the pixel of concern are input to a median filter. While 
median hybrid has been shown to work quite well in 
discriminating some edges, it is deficient in several respects 

n..'fU m ™„,,J * n — 1 1 - 4. i* t-u - jr. _ nt . _ _ .1 . .. 

..w uvivvnuu. J. no insula 11 11 v L>1 1U 111 LOl UUCd 

not consider the noisiness of the edge itself In other words, 
an edge's direction, even though eight are employed, cannot 
be determined with exacting accuracy. For instance, an edge 
feature may lie at a 33 degree vector from a particular pixel, 
and thus the eight directions are inadequate in determining 
the edge feature. In other words, a single pixel may contain 
a portion that is edge and a portion that is non-edge in the 
non-discrete world that cannot be represented in the discrete 
world of digital images. When applied to digital images, the 
median hybrid filter, if applied everywhere to all pixels, may 
propagate noise or shift it from pixel to pixel while attempt- 
ing to remove it since there is noise along the edge feature 
due to the non-cardinal direction of the edge. A curved edge 
is a perfect example of such a problem. 

When an object/scene is imaged by a sensing or imaging 
device such as a digital camera, the resultant image in 
captured into a CFA (Color Filter Array) bearing a particular 
color channel pattern. One oft -used pattern for capturing 
images is known as the Bayer pattern, which has color 
channels as follows, 

GRGRGR... 

BGBGBG... 

GRGRGR... 
with rows thereafter repeating the pattern. 

Thus, in a Bayer pattern CFA, each pixel location has an 
intensity value associated only with one of the three color 
planes (Red, Green and Blue) which combine to make a full 
color. The process of estimating the two missing color 
components for each pixel location is known in the art as 
color interpolation. The interpolation of color often precedes 
the removal of noise in color images due to the fact that most 
traditional noise reduction or removal techniques are 
designed to operate upon images with full color pixel 
information. The process of color interpolation itself will 
introduce noise, and in the case of a digital camera, where 
the color interpolation is most likely performed after image 
compression and download to a data processing system such 
as a computer, the intermediary steps of color resolution 
companding, compression (quantization and encoding) and 
decompression may add additional noise such that the 
original captured image noise may be blended with other 
noise to perhaps lose the distinction of being noise and gain 
the distinction of being an image feature. Performing noise 
removal on the full color pixels attained by the color 
interpolation process increases the memory and processing 
needs of the noise removal process by a factor of 3 (since 
each pixel has thrice the resolution), and thus is difficult and 
expensive to implement in hardware. Other noise removal 
techniques attempt to reduce this burden by performing 
color space conversion after color interpolation into, for 
instance, the YUV space, where only the Y (chrominance) 
component is considered for noise removal. However, this 
too may propagate additional noise beyond that propagated 
by color interpolation and also cannot be easily be imple- 
mented in hardware. 

Thus, there is a need for a noise reduction framework that 
will not only distinguish edge pixels from non-edge pixels, 
but also one that can work directly in the CFA image domain 
prior to any color interpolation. 

SUMMARY OF THE INVENTION 

A method is disclosed having the steps of classifying 
pixels of a captured image while in its Color Filter Array 
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form as either edge pixels or non-edge pixels. The method tional difficulty since the noise patterns are combined in an 

next performs noise removal by applying a first noise unknown way. The resulting image will have elements that 

removal technique to those pixels classified as non-edge cannot be distinguished as noise, and thus, when the Bayer 

pixels and applying a second noise removal technique to pattern CFA is subjected to color interpolation, as is required 

i1 _1 ZJl . -1 - ,1 „: 1_ T-l _ .1. ; „_ I i 1_ _ Q r*rir\ri -fVi-r mncf nnicn rom rr. 1 ta/ikm'/iiicr *lm svtAn^n 

Liiisa*^ v^iajsmvu aa v^u^ piAVLS. uulii ihjicm* iwiiuvoi ivwu- j — r ■ **- *n j — 

niques are applied while the image is still in the Color Filter bleec J from edge t0 ed S e and feature to feature. For instance, 

Array form. consider a blue edge feature lying on a yellow background. 

Due to noise a red pixel may lie somewhere on the edge 

BRIEF DESCRIPTION OF THE DRAWINGS boundary. Due to typical color interpolation methods, the 

blue and yellow will be averaged together with the red to 

The objects, features and advantages of the method and form a full pixel with blue, yellow and red somewhere in the 

apparatus for the present invention will be apparent from the neighborhood (since red appears to exist as a feature, albeit 

following description in which: small, within the image region), and thus, a false color pixel 

FIG. 1 is a flow diagram on one embodiment of the will be generated which may not thereafter be discernible as 

invention noise to a noise removal process. 

cr^ ->/ \ mi ♦ + i t u* • • « * j- *• 35 Thus, it would be desirable to devise a noise removal 

FIG. 2(a) illustrates a mask for obtaining a first direction ■ 4 t,„* „ T _i_ ■ fU pea j • • ♦ *u 

diff vy & technique that works in the CFA domain prior to the process 



of color interpolation so that the additive noise and color 



FIG. 2(6) illustrates a mask for obtaining a first direction bleeding problem can be circumvented. Further, the tech- 
differential, nique should treat and consider pixels belonging to edge 

FIG. 3 illustrates the application of he noise removal 20 features in the image differently from pixels not belonging 

technique independently in each color plane according to t0 ed S e features. Also, the technique(s) used should be 

one embodiment of the invention. capable of fast hardware implementation such that it can be 

™« * *n a * ,i • ■ i r .i l j- implemented, if desired, in a digital camera where the CFA 

FIG 4 illustrates the pnnciples of at least one embodi- ^ rated 0ne such possible ^ oise removal met hodology 

ment of the invention as applied to an exemplary image ^ is shown in FIG x and in various Qther emb odiments 

re 8 lon - discussed below. 

FIG. 5 is a block diagram of a three-level median hybrid FIG. 1 is a flow diagram on one embodiment of the 

filter as employed in the various embodiments of the inven- invention. 

tl0n * First, an initial localization region within the captured 

FIG. 6 is a block diagram of a three-level median hybrid 30 image is defined (step 110). The initial localization region 

filter as employed in the various embodiments of the inven- will be a M by N rectangular region of the captured image 

tion. where M and N may be equal if desired thereby defining a 

FIG. 7 is a system diagram of one embodiment of the square region. The localization region as a percentage of the 

invention, tota ^ ^P^ed image size may be large or small depending 

35 upon the desired implementation and available hardware/ 

DETAILED DESCRIPTION OF THE software resources and speed constraints for how fast the 

INVENTION image is desire to be processed for the removal of noise. 

Where the image is more uniform in characteristics over a 

Referring to the figures, exemplary embodiments of the i arger reg ion, the localization region may be defined large, 

invention will now be described. The exemplary embodi- 4Q However, if fine detail discrimination is crucial or the image 

ments are provided to illustrate aspects of the invention and shows w j de variations in characteristics within small 

should not be construed as limiting the scope of the inven- regions, the localization region can be defined smaller, 

tion. The exemplary embodiments are primarily described 0nce the localization region fe defined) the next step is t0 

with reference to block diagrams or flowcharts. As to the det ermine the gradient value associated with each and every 

flowcharts, each block within the flowcharts represents both ^ pixel in the localization regiorl by applying some mask or 

a method step and an apparatus element for performing the operalor (step 120) ^ mask or gradiem operalor is 

method step. Depending upon the implementation, the cor- applied to a small neighborhood about each pixel, ordinarily 

responding apparatus element may be configured in with that neighborhood as well being of smaller size than the 

hardware, software, firmware or combinations thereof. localization region. FIGS. 2(a)~2(b) and associated descrip- 

When an image captured by a sensor or camera or other 50 tion illustrate the parameters and configuration of such a 

imaging device is analyzed, it can be said to have portions mask with a 3 by 3 neighborhood about and including the 

of the image which are "noise" or "noisy" while other pixel whose gradient is being determined. By repeatedly 

portions are not. The determination of what constitutes noise applying the operator or mask to each pixel in the localiza- 

is difficult and not an exaction process. Noise may be tion region and determining the gradient thereby, gradient 

confused with a detail such as thin line or "edge" portion of 55 information for the localization region can be completed. In 

the image. The removal of noise in a captured image is most gradient-based edge detection, the gradient value is 

optimal when detail information is still preserved while normalized over the entire image by dividing the gradient by 

noise is removed. In traditional noise removal, each pixel, the maximum gradient in the entire image. However, such a 

whether having edge features or not, is treated by the same normalization would require, if implemented in hardware, a 

noise removal technique due in large part to this relative 60 two-pass approach where gradient information for all pixels 

inability to distinguish noise from edges. in the entire captured image must be determined and then, 

In color images, such as those captured into a Bayer each gradient divided by the maximum gradient value found 

pattern CFA (Color Filter Array), the step of color interpo- within the image. In the various embodiments of the 

lation may additionally introduce noise and thus, the original invention, the gradient may instead be normalized by the 

noise due to the imaging device itself is further perturbed. 65 maximum within the localization region. 

The addition of various types of noise, at each stage of the Once the gradient information is determined, a threshold 

imaging process from capture to rendering presents addi- should be selected (step 130). The threshold value may be 
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selected as a single value applicable throughout the image or Preferably and for a fast implementation, the localization 

as varying from localization region to localization region. regions should be exclusive, but may overlap if desired. The 

Further, the threshold may be preset to a particular value for noise removal technique described in FIG. 1 may be applied 

?i, Very J Cap t mred by the ima § in S device * Such a P rcset directly in the CFA domain to pixels in Bayer pattern prior 

— • ~"J — • — ■ - r -~- — 5 iu any cuiui iLitcipvjid.uuii. Tuus, iiic FIG. 1 iccuuiuuc, if 

detection process . but according to the current state of the employed, will reduce the amount of propagated noise 

art, is mostdesirable for a hardware implementation of noise throughoul the various st of process i D g the image, 

removal. The selection of this threshold is not a subject of nT ° . . & & 

this invention, but if the gradient information of every pixel . FIG * 2 S a > lllustrates a ma sk for obtainmg a first direction 

within the localization region is first determined it may be 1Q differential. 

possible to dynamically select a threshold value. One means of determining the gradient for a pixel is to 

Whatever the selected threshold value, the gradient (or a PP lv a mask A which essentially finds the "strength" of the 

normalized gradient) is compared against that threshold P ixel about a neighborhood boundary that the mask covers, 

value (step 140). If the gradient (or normalized gradient) Illis strength is essentially a differential in a particular 

exceeds the threshold value, the corresponding pixel can be 15 direction along two boundaries of a pixel. This first 

classified as an "edge" pixel which is a pixel that belongs to direction, when combined with a similar computation for the 

a edge feature of the image such as a line. If not, the pixel strength along a second direction boundary gives a good 

is classified as a non-edge pixel. According to one embodi- estimation of the relative strength of the center pixel as 

ment of the invention, the edge pixel and non-edge pixel against its neighbors. In FIG. 2(a), 3 by 3 mask is illustrated 

each receives a different noise removal treatment. While one 2Q tnat has non-zero coefficients in its 1 st and 3 rd columns, and 

method of edge detection, namely gradient based edge zero coefficients in the middle or 2 nd column. The mask A 

detection has been presented in the flow diagram of FIG. 1, * shown to have six non-zero coefficients Al, A2, A3, A4, 

any suitable edge detector may be utilized. For instance, the ^ and A ^ When the mask is applied to an Image region I 

well-known Canny edge detector, which uses Gaussian having pixels with intensity values 10, U, 12, 13, 14, 15, 16, 

masks convolved with the image space in classifying edge 25 17 and I8 » tne resultant differential Ax is expressed as: 

and non-edge pixels, may be used in conjunction with the Ax-Al*I0+A2*I3+A3*I6+A4*I2+A5*I5+A6*I8. The dif- 

technique of FIG. 1, and'if capable of implementation in the ferential Ax can be used in calculating the gradient of the 

desired application, will provide good edge discrimination. P ixel witn inten siLy value 14, shown to be in the center of the 

For a digital imaging device such a portable still digital image region L Ordinarily, the mask A will have coefficients 

camera, an edge detector such as the one presented in FIGS. 30 tnat summed together give zero. In the well-known Sobel 

1, 2(a) and 2(b) are more optimal due to the speed of their ma f k > the coefficients Al, A2 and A3 all have values of -1 

computation and use of less silicon area within the device. wmle lne coefficients A4, AS and A6 all have values of +1. 

By classifying pixels as either edge pixels or non-edge with a Sobel mask ' s coefficients, Ax, the differential, is 
pixels, they may, according to one embodiment of the Ax=I2+I5+I8-I0-I3-I6. Thus, the choice of mask 
invention, be then subjected to different noise removal 35 coefficients, which is known in the art to achieve a mean- 
technique and treatment. Thus, according to step 160, a mgful gradient value, simplifies the differential computation 
noise removal technique such as multi-level median hybrid to a mere addltlOD and subtraction of pixel intensity values 
filtering may be applied upon the pixel (see FIG. 5 and whlch can be implemented easily in hardware by adders, 
associated description) if the pixel is a non-edge pixel. This lllus » lhe differential becomes easy to implement and fast. A 
treatment avoids the problem posed by median filtering of 40 dlfferenl mask ' sucn the Prewitt mask, if applied can be 
blurring and decimating edge features, since the pixel does as easil y implemented. The Prewitt mask has coefficients of 
not belong to an edge feature. The median filtering, when A } and M ec l ual to _1 and an ^ coefficient of -2. To 
applied to the non-edge pixel will have the desired effect of mirror » the Prewitt mask has coefficients on the other direc- 
removing noise without blurring the edge tlonal boundarv °f A4 and A6 equal to +1 and a coefficient 

Since noise may also be present in a pixel that is classified 45 £* ° f +2 A Usi " g J J£™f 1 ™^V? e ^ ° T differential value 

as an edge pixel, the edge pixel is subjected to a smoothing hcC0 ™l 15+18-10-2*13-16. Though a multiplica- 

technique such as linearly averaging neighboring pixels ^ by 2 is introduced, this can be implemented easily by a 

having a similar intensity value (step 150). This smoothing S A mft T'tffc °{J 5 ™ l 13 ° r the computation 

ensures that an edge is not completely decimated and in fact! AX * S Ax = I2+ I5-hI5+I8-I0-I3-I3-I6 if only add and subtract 

may enhance the edge in appearance (i.e., sharpen the edge 50 °P eratl0ns are implemented in the hardware, 

by smoothing together the color (intensity) more uniformly). The gradient operation is used in edge detection since it 

The linear averaging technique replaces the pixel under I s ^sumed that an edge has a particular direction within the 

consideration with a linear average of itself and neighboring ™ a S e space. The pixels along the side boundary (one 

pixels that have a similar intensity. One method of discov- side — N>> 13, 16 and another side— 12, 15, 18) of the pixel 

ering which pixels qualify to be of "similar intensity" is 55 should have a strength or relative combined intensity level 

described below with respect to FIG. 6 and its associated less lnan tne P ixel 14 itself if 14 is to be considered an edge 

description. Essentially, the goal is that two or more edge pixel. In tnis regard, a first direction differential, essentially 

features that border each other should not be confused when ^ tne horizontal direction (the mask has values determining 

averaging so that edge features are not combined together lne strength on the left and right sides of the image) may be 

improperly. 60 computed as given above. Since the image space is in two 

Next, if any pixels are remaining to be treated to noise d imensions, vertical and horizontal, only one more direction 

removal within the localization region (checked at step 170), differential about the pixel 14 need be computed, 

the next pixel's information is fetched (step 175) and steps FIG. 2(b) illustrates a mask for obtaining a second direc- 

140-160 are repeated. Once all pixels within the localization uon differential according to one embodiment of the inven- 

region are treated to noise removal, a new localization 65 Xl0n < 

region may be defined (step 180). Processing then continues In FIG. 2(6), a second 3 by 3 mask is illustrated that has 

with steps 120-170 repeating for the localization threshold. non-zero coefficients in its 1 st and 3 rd rows, and zero 
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coefficients in the middle or 2 nd row. The mask H is shown 
to have six non-zero coefficients Bl, B2, B33, B4, B5 and 
B6. When the mask is applied to an Image region I having 
pixels with intensity values 10, II, 12, 13, 14, 15, 16, 17 and 
IS the resultant different:-! A" iz er""c c,n d oc* A "- D1 * Tft i 
B2*Il+B3*I2+B4*I6+B5*I7+B6*l£ The differential Ay 
can be used in calculating the gradient of the pixel with 
intensity value 14, shown to be in the center of the image 
region I. Ordinarily, the mask £ will have coefficients that 
summed together give zero. In the well-known Sobel mask, 
the coefficients Bl, B2 and B3 all have values of -1 while 
the coefficients B4, B5 and B6 all have values of +1. Thus 
with a Sobel mask's coefficients, Ay, the differential, is 
AyoI6+I7+I8-I0-Il-I2. Thus, the choice of mask 
coefficients, which is known in the art to achieve a mean- 
ingful gradient value, simplifies the differential computation 
to a mere addition and subtraction of pixel intensity values 
which can be implemented easily in hardware by adders. 
Thus, the differential becomes easy to implement and fast. A 
different mask, such as the Prewitt mask, if applied can be 
as easily implemented. The Prewitt mask has coefficients of 
Bl and B3 equal to -1 and an B2 coefficient of -2. To mirror, 
the Prewitt mask has coefficients on the other directional 
boundary of B4 and B6 equal to +1 and a coefficient B5 of 
+2. Using a Prewitt mask, the Ay or differential value in the 
second direction along the image boundary becomes Ay«I6+ 
2*17+18-10-2*11-12. Though a multiplication by 2 is 
introduced, this can be implemented easily by a shift opera- 
tion of 15 and 13 or by splitting the computation Ax as 
Ay=I6+I7+I7+I8-I0-Il-Il-I2 if only add and subtract 
operations are implemented in the hardware. Preferably, the 
mask A used to find the Ax or first direction differential 
should be the same type of mask H that is used to find the 
second direction differential. It may be desirable to use a 
Sobel mask in one direction and Prewitt in another to 
compensate since the imaging device may not yield square 
pixels, but rather, rectangular pixels or pixels of other shape. 
In such a case, the geometry of a pixel may be accounted for 
in the choice of mask coefficients. 

With the second direction differential Ay obtained as 
given above, both directional boundary pairs horizontal and 
vertical are accounted for. Then the gradient of 14, the 
central pixel in the image space I is the magnitude of the 
vector obtained by a vector which is the sum of the vectors 
obtained i n each direction. Hence, the gradient G is simply 
VA x 2 +Ay 2 , or the magnitude of that vector formed by 
combining Ax and Ay. The gradient of each pixel may thus 
be obtained by considering a neighborhood, preferably a 
small 3 by 3 neighborhood that surrounds each pixel, and 
then by applying a mask upon it in each of two directions 
and their corresponding boundary pairs essentially thereby 
including every immediately adjacent pixel in the gradient 
determination. As mentioned above, the gradient value may 
be normalized for the maximum gradient present in the 
image so that relative weight is given to the gradient 
measure. For instance, in a captured image of a scene with 
more texture rather than hard edges such as lines, the 
gradient values may lie within a narrow region. If these are 
normalized, to the maximum gradient, then the gradient 
values obtained can be more effectively analyzed. 

FIG. 3 illustrates the application of the noise removal 
techniques to each color plane independently. 

Since the noise removal technique presented in various 
embodiments of the invention operate directly upon the CFA 
version of the captured image which is essentially the raw 
captured image, it may be desired in one embodiment that 
the color planes may be separately considered. Thus, con- 
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sidering the 6 by 6 pixel Bayer pattern region of the image, 
the Blue (designated by "B" in FIG. 3) color plane (i.e., 
pixels associated with the color blue) may be separated out 
from the CFA and have the noise removal technique 

_ A^r^riu^A „u rt T^ n « A i: ar ) «~ ;+ ♦ — t — i i t*i 

the pixels associated with the blue plane at locations (2,1), 
(4,1), x(6,l), (2,3), (4,3), (6,3) and (2,5), (4,5), (6,5) can be 
arranged so that they are each adjacent one another. 
Likewise, other neighboring pixels in the Blue plane (not 

1Q shown) of the CFA may be combined with the nine pixels 
illustrated to form a Blue sub-image. The noise removal 
framework described in the various embodiments of the 
invention may then be applied to this Blue sub-image 
directly and independently or other color planes. 

In the Bayer pattern, Green pixels have roughly twice the 

35 representation of either Blue or Red pixels in the CFA. 
Often, it is useful to split the green pixels into those that 
accompany a row of alternating Blue pixels (a first Green 
labeled Gl) and those that accompany a row of alternating 
Red pixels (a second Green labeled G2). Thus two Green 

20 color planes, Gl and G2 may be extracted from the CFA. 
This is useful since the number of computations for each 
color plane will then be identical. If Gl and G2 planes were 
considered together when applying the noise removal tech- 
nique thereon, there would need to be a row and column 

25 offset to combine the Gl and G2 pixels (since they are on 
different rows and columns), and further the combined 
sub-image of a unified Gl and G2 would be twice as large 
in pixel count as the Blue and Red sub-images, leading to the 
need for a different, larger sized memory. For the purposes 

30 of analyzing the image, it is useful to consider the two Green 
color planes separately since the Green is vital in determin- 
ing the chrominance (color content) present in neighboring 
pixels. This fact can be of use in techniques such as color 
interpolation or color resolution companding. To preserve 

35 the chrominance information in a more effective way for 
later use, therefore, it I desirable to consider the Gl and G2 
color planes separately for the purpose of noise removal. If 
the noise is removed from the Gl and G2 sub-images 
independently rather than together according to the various 

40 embodiment of the invention, then the noise removal will 
more accurately reflect the relationship of details to their 
original color. The edge features have direction, and a 
combining of Gl and G2 pixels, which always lie at a 45 
degree offset from one another, would introduce a 45 degree 

4S shift in the orientation of the edge. 

FIG. 4 illustrates the principles of at least one embodi- 
ment of the invention as applied to an exemplary image 
region. 

FIG. 4 shows a 6 by 6 pixel region of a captured image. 

50 A pixel x at a row location "i" and column location "j" will 
be represented hereafter as x(i,j). The exemplary image 
region shows both pixels that have already been classified as 
edge pixels (designated by an "E") and non-edge pixels 
(designated by an "NE"). The edge pixels are further des- 

55 ignated by subscripts indicating which edge feature the edge 
pixel belongs to. For instance, a first darker edge feature 
(designated by E 2 ) may have intensity values in the neigh- 
borhood of 200, while a second lighter edge feature 
(designated by E 2 ) may have intensity values in the neigh - 

60 borhood of 50. Each of pixels E-, and belong to two 
different edges of the image. The non-edge pixels have not 
been sub -classified into feature, and the classification of 
edge feature for edge pixels shown in FIG. 4 is merely to aid 
explanation and is not a limitation or requirement of the 

65 invention in any embodiment. 

According to one embodiment of the invention, the edge 
pixels and non-edge pixels within an image are subjected to 
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different noise removal techniques. In one embodiment, noise removal technique that utilizes a multi-level median 

pixels that are not classified as belonging to an edge (NE in hybrid filtering process. A three-level median hybrid filter is 

FIG. 4) are applied upon by a first noise removal technique. designed for a K by K block about an input pixel x(ij) that 

In one embodiment of the invention, this first noise removal uses linear averaging subfolders to obtain the various values 

don of such filtering is detailed below with respect to FIG. stage or level of the flIter shown - m FIG 5 ^ if K=5 a 

5. All pixels classified as edge pixels are subjected to a 5 b 5 neighborhood about and including the pixel x(ij) is 

second noise removal technique different from the firs and considered for median h brid filtering. According to one 

having better edge preservation/enhancement characteristics ~e .v.. .u j- u u j c 

than the first technique. In one embodiment of the invention, „ !5*°* men of the mvention, the median hybrid filter of 

this second technique, applied to non-edge pixels, is a linear^ 10 F ]?■ 5 ^ appbed to the pixel x(ij) only if the pixel 

averaging of adjacent pixels having similar intensity values x(lJ j 600,6 be classified 35 an non-edge pixel, according to 

as the pixel under consideration. ^ adient ° r «*»nnition. Otherw.se, if pixel x(ij) were a 
r- • . c . i-i»/n( . . , ™« edge pixel, the nearest neighboring pixel averaging method 

For instance, referring to FIG. 4, the pixel NE(3,3) A 1 1 %u , < a ul i- j 

f A . i i i\ UL j • t discussed below with respect to FIG. 6 would be applied, 

(non-edge pixel at row 3, column 3) would be treated with Y a Hjy n^ u . 

the median hybrid filtering technique, and is described in The subfilters (not pictured) simply take the 

detail below with respect to FIG. 5. The output NE'(3,3) of avera S e of a number of neighboring pixels as desired by the 

the median filtering process would become the new noise Mze ° f the nei S hborhood - The averaging subfilters are 

removed pixel at location (3,3). The median filtering process ^ h f d t0 * vector of P lxels / n each of ^ eight directions 

would utilize all the pixels, whether they are classified as on (no * h ' east ' west > northeast > southeast, southwest and 

non-edge pixels or edge pixels, within the neighborhood n ° r ^ est >; Hen ° e ' the * VGT *® n Z subfilter s would produce 

(from pixel location (1,1) to (5,5) about the pixel NE(3,3) the followin S 0Ut P ut val » e s Y^ C7roA (ij) ™ follows: 

when performing its noise removal process. Y £^57<io)=(x(io+l)+x(io +2))/2; 

By contrast, the edge pixel Ej(3,2) would be applied upon Y WES7 (ij)=(x(i j-l)+x(i,j-2))/2; 

by a linear averaging of only other edge pixels of similar 25 Y sour/f(io) := (x(i+lj)+x(i+2j))/2; 

intensity (edge pixels belonging to the same edge feature) to Y NORTfJ (i,y)=(x(i-l j)+x(i-2,j))/2; 

remove the noise present in £^3,2). Of the eight pixels Y 50 ^ jHEASJ <i,j)=(x(i+lo+l)+x(i+2,j+2))/2; 

adjacent to £^3,2), only three classify as belonging to the Y 50OT - mKFiS7 <ij)=(x(i+l 3 j-l)+x(i+2o-2))/2; 

first edge feature, namely, pixels £,(2,2), £,(4,2) and E,(4, Y^^^oXx^-lj+^+xG^j^; and 

3;. ITius, the linear averaging noise removal technique ™ 1 v J\ \\-(^(\ 1 i \\^(\ o ; own- 

would average together four pixels (three adjacent and the ■ ^^fil^^ fllters 

P h X p ? 3 > 2 )' H 2 ®. Ei(4, along with the ce nter pixel value x(i J} variousl as shown w 

V ul ^ ^ i f h cond i ealur ^ '* should ? e FIG. 5. By using an averaging sub-filter, the median opera- 

noted that the pixel 12,(6,2) has three adjacent edge pixels, , ion fa reduced lo a ^ f rf three y ^ s fa J f 

but only one edge pixel E 2 (6,l) that belongs to the same 35 lhe median h brid filterin ocess 6 

edge feature (i.e., has a similar intensity value). The other F , a 5 shows ma , , he ^ levd of 

the three-level median 

Kvo edge pixels El (5,3) and El (6,3)that are adjacent pixel hybrid fflter faas a ^ of median fllters 510 512 5M and 

E 2 (6,2) belong to a firs edge feature, and thus should not be 516 Median filter 510 has m t M1 ^ ^ ^ median 

averaged with be pixels of the second edge feature. Other- of three yalu namel Yea J^ YwESI (ij) and x(i,j). 

wise the second edge feature would be decimated with its 40 Median flUer 512 has ^ ou ^^j t^lf the median of 

intensity value contaminated bya non-relevant and cussimi- ^ val Mmel y^^), Yso£nH (y) and x(i,j). 

lar intensity vMue or values. The me^odology shown in Median fllter 514 ^ J^xMi iha^fethe median of 

FIG. 6 and detailed in the associated description below , hree values namel y^^^ij), Y SOUTftWE Ui,j) and 

circumvents this sUuation by discerning ; edge pixels of x(i<j) . pj^ mcd J' £g ^ ^ I ou^WWuit i the 

similar intensity from other edge pixels. FIG. 4 shows a third 45 median of three values name , Y NOXTffWESJ (i,i), 

edge feature (havmg pixels designated E 3 ) and fourth edge Y n . nA vfi ; , " northwest\ yjj< 

feature (having pixels designated E 4 ) which should be as £ Second tel, there are a set of two median 

carefully treated to avoid contamination from dissimilar filters 520 and 522 wfaich ^ ^ ^ ^ median ^ 

edge pixels. outputs Ml, M2, M3 and M4 and the original pixel value 

FIG. 4 shows only a portion of the image, but with regard 50 x(i j), Median filter 520 provides an output M5 that is the 

to pixels lying on the physical edges of the image (first me dian of Ml, M2 and x(i,j). Likewise, median filter 522 

column, first row, last column, last row, etc.) these may provides an output M6 that is the median of M3, M4 and 

simply not be processed for noise removal since a portion of x(i j). 

the mask(s) will have no values, and thus, ineligible gradient At * the final or third level> a single median filter 530 

informaUon would be attained thereby. Alternatively, it may 5S pr0 vides the output pixel x'(id) that is the median of the 

be desirable to use the dark current or reference pixels (of outputs of the median filters from the levdj M5 and 

which there are usually several rows and columns) to M6, when combined with the original pixel x(ij). Thus, the 

substitute for missing values for edge pixels. These dark noise removed pixel x . (i j} ^ equal to the median of M5 M6 

current pixels are not considered part of the final image CFA and x(i j). ^ noise remoV ed pixel will, in one alternate 

generated by the imaging device but are nonetheless cap- 60 embodiment, be stored in a separate array to not interfere 

tured by the sensing device to aid in determining pixel with the rema inder of the noise removal process for other 

v pixels. In this manner, the original pixel space is maintained 

FIG. 5 is a block diagram of a three-level median hybrid so that the noise removal process may be iterated and 

filter as employed in the various embodiments of the inven- adjusted to provide better results. In another alternative, the 

tlon * 65 noise removed pixel will replace the original pixel imme- 

According to one embodiment of the invention, a pixel diately so that the noise removal process is recursive. This 

x(i,j) if classified as a non-edge pixel is applied upon by a has the advantage of speed over an iterative method but also 
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introduces uncertainty as a misclassified pixel will tend to A^ 0 ^ r// =|x(i,j)-x(i-lj)|, A yo[/77/ «|x(ij)-x(i+l j)| 

propagate its mischaracterization to affect other pixels. A H ^ 5r »|x(ij)-x(ij-l)|, A £A5r =|x(ij)-x(ij+l)| 

Thus, the on-the-fly compute and replace method is not ^northeasMA^})-^- 1 ^-^* A A c WjtfWfe - i - y «|x(i,j)-x(i+ 
guaranteed to provide good results, though in some cases it 

may improve the noise removal. a ^nortnwesj^(^)-^-^ +1 % ^southeast^^})-^ 

Referring back to the example of FIG. 4, the non-edge 1 J— 1)1* 

pixel NE(3,3) would, according to one embodiment of the 0nce a11 ei S ht AVDs ( one for eacD adjacent pixel in the 

invention, be subjected the three-level median hybrid filer. ei S ht cardinal directions) is determined, then the AVD 

As such, Y^ y <3,3)=(NE(3,l) + E 1 (3,2))/2 and ¥^3,3)= av f ra S c value 15 determined (step 610). The AVD average 

(NE(3,4) + E 4 (3,5))/2. Likewise, the other averaging subfil- 30 val || e 15 SUD ^ the sum of lhe dividual AVDs divided by 

ters can be applied about the location (3,3) to obtain the ei8 Jr t " iL A _._ . . J • _, , 

u^n * *■ t*i_ u ii_ * « ■ , ,i Once the AVD average value is determined, the next step 

various subfilter outputs. The results are then fed into the , , , iU r . B *. t . , * . . F 

A n U • #u • i i i u KTr/( , ^ . is check whether a first adjacent pixel is an edge pixel or not 

median filters in the various levels along with NE0.3) at ( 6U) , f ^ fc n J ot an £ ^ ^ g ^ be 

Ca ? ™ m ,f u f S ' Dt mtenSlty 15 suited for the av Wg method and is excluded from the 

value NE (33) becomes the noise-removed version of the pixel averagmg xi (step 640). If the adjacent pixel is an edge 

original captured/decimated pixel NE(3,3). pixel> then another compari son for relative correlation is 

The three-level median hybrid filter will essentially performed to check whether the adjacent pixel belongs to the 

closely preserve the original non-edge pixel value because same edged feature as the pixel under consideration, or 

the median at each stage or level of the filtering process 2 0 belongs to different edge feature (see FIG. 4, showing pixels 

re-introduces the original pixel x(i j) as one input at each of E 1 , 1^, E 3 and E 4 belonging to four different edge features 

the median filters. Though the three-level median hybrid of varying intensity. The test for correlation (step 615) is 

filter has been shown to work well with most images, performed upon discovering that the adjacent pixel is an 

applying it universally to all pixels of an image is disad- edge pixel. According to step 615, the adjacent pixel's AVD 

vantageous. The orientation of the edge may not be 25 is compared against the average AVD value determined at 

discovered, as mentioned above, and further depending upon step 610. If the adjacent pixel is not correlated, i.e., its value 

the signal-to-noise ratio of the image, a noisy edge pixel may is not less than the average AVD, then the pixel is excluded 

be passed through the median filter unabated. from the averaging set (step 640). 

FIG. 6 is a flow diagram illustrating nearest neighbor If the adjacent pixel is an edge pixel and its AVD is less 

averaging for edge pixels according to one embodiment of 30 than the average AVD, then the pixel is included into the 

the invention. averaging set which is utilized for the noise removal tech- 

In one embodiment of the invention, an edge pixel is nique (step 620). As part of the averaging set, the first step 
treated to a different noise removal technique than a non- is to add the intensity value of that adjacent pixel to a 
edge pixel. Specifically, an edge pixel needs to retain the variable (register) ACCUM which essentially keeps a run- 
values of neighboring pixels that are also edges. When two 35 ning sum of each averaging set pixel (step 625). The 
different edge features are adjacent and have pixels adjacent ACCUM variable or register will facilitate the averaging 
each other (for instance, FIG. 4 shows edge pixels E 2 (6,2) determination (performed at step 670). Since it is unknown 
and E 2 (6,3) belonging to a second and first edge feature at the beginning of the noise removal by averaging tech- 
respectively), then it is necessary to discriminate between nique for edge pixels how many adjacent pixels are within 
them so that the wrong intensity value is not averaged 40 the averaging set, a second variable (register in hardware) 
together with the pixel under consideration. Thus, it is NUMCOUNT is used to track the number of pixels within 
desirable to average together neighboring (adjacent) pixels the averaging set. Every time a new adjacent pixel is 
that are both edge pixels and have "similar" intensity values included in the averaging set, NUMCOUNT is incremented 
with the pixel considered for noise removal. Intensity values by one, starting at an initial value of zero, 
are determined to be similar to by comparing the absolute 45 If any other adjacent pixels are remaining to be consid- 
value difference between the pixel under consideration for ered for averaging, then the next adjacent pixel is fetched 
noise removal and adjacent pixel with the average of all such (step 645) and the steps 612 through 650 repeat. When a 
absolute value differences between the pixel under consid- pixel is excluded from the averaging set, the same check for 
eration for noise removal. If the difference is less than the other remaining pixels (step 650) is performed. When there 
average, then the adjacent pixel is considered to be of 50 are no more adjacent pixels left to be considered for the 
"similar" intensity with the pixel under consideration for averaging set,, then finally, the value of the pixel x(ij) under 
noise removal. consideration for noise removal is added to ACCUM (step 

FIG. 6 illustrates a flow diagram to perform such nearest 655). This facilitates the averaging in of the original pixel 

neighbor averaging of similar intensity pixels. First, accord- value with the averaging set so that the original pixel value 

ing to step 605, the AVD (absolute value difference) or 55 has equal representation therein. In addition, NUMCOUNT 

distance between the pixel x(i j) and each adjacent pixel is is incremented by one (step 665) so as to account for the 

determined. For instance, the north neighboring adjacent extra pixel x(i j) included in the averaging set. According to 

pixel x(i-l j) has a distance A^ 0/OTy -|x(ij)-x(i-l j)|. step 670, the average of the pixels in the averaging set is 

Likewise, the AVDs of each adjacent pixel to the pixel x(ij) determined by dividing the contents of ACCUM by NUM- 

is determined. The AVD is essentially a measure of corre- 60 COUNT. The resultant average is the noise removed pixel 

lation between the pixel under consideration for noise x'(i,j). This value x'(i,j) is stored into an array or table so that 

removal and the adjacent pixel which is being considered for the original value x(i,j) is not overwritten and can be used in 

inclusion into the averaging set (see step 620). Those further noise removal computation for other pixels (step 

adjacent pixels that are well correlated linearly with the 675). When the noise removed pixel value x'(i,j) is 

pixel x(i j) will be averaged together with the value x(i j) to 65 determined, the variables (registers) ACCUM and NUM- 

attain the noise removed output pixel x'(ij). The AVDs thus, COUNT are cleared (set to zero to facilitate the processing 

are: of the next edge pixel x. When the noise is thus removed 
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from the edge pixel x, another pixel classified as an edge devices, buses and bridges can be utilized with the invention 

pixel may be fetched (step 685) and the steps 605-680) and the combination shown is me rely illustrative of one such 

repeated for that new pixel. possible combination. 

By averaging together adjacent edge pixels of similar When an image, such as an image of object/scene 740, is 

intensity witn the originai pixel xjij), the edge feature is i captured, ihe image is sensed as k, G, and 5 pixels, in a CPA 

maintained and may even be enhanced due to the smoothing such as the Bayer pattern. These pixel values are sent to the 

and uniformity across the edge as a result of averaging. image processing circuit 732. Image processing circuit 732 

Treating the edge pixels differently from non-edge pixels consists of ICs and other components which execute among 

avoids the traditional problems of noise removal edge deci- other functions, an image compression scheme to reduce the 

mation and bleeding in color after interpolation is per- 10 size of the transfer between camera 730 and computer 

formed. The averaging technique may also be implemented system 710. The need for color interpolation is most sig- 

in a fast hardware architecture as desired in portable digital nificant when the image is to be displayed or output and thus 

camera, and can apply directly to the CFA domain (Bayer does not have to be performed on camera 730. When the user 

pattern RGB color space prior to color interpolation). or application desires/requests a download of images, the 

FIG. 7 is a system diagram of one embodiment of the 15 compressed images stored in the image memory unit are 

invention. transferred from image memory unit 734 to the I/O Port 717. 

FIG. 7 illustrates a computer system 710, which may be I/O port 717 uses the bus-bridge hierarchy shown (I/O bus 

any general or special purpose determining or data process- 715 to bridge 714 to system bus 713) to temporarily store the 

ing machine such as a PC (personal computer), coupled to image data into memory 711 or, optionally, to disk 718. The 

a camera 730. Camera 730 may be a digital camera, digital 20 compressed images are decompressed by suitable applica- 

video camera, or any image capture device or imaging tion software (or dedicated hardware) and may utilize pro- 

system, and is utilized to capture a sensor image of an object cessor 712 for doing so. The decompressed image data will 

or scene 740. Essentially, captured images are compressed be recovered as a 8-bit Bayer pattern CFA similar or 

and processed by an image processing circuit 732 so that identical to the image data as captured by the sensor of 

they can be efficiently stored in an image memory unit 734, 25 camera 730. 

which may be a RAM or other storage device such as a fixed The downloaded image once decompressed will thus have 
disk, miniature card, etc. In most digital cameras, raw pixels in their raw CFA form. If a noise removed image is 
images obtained from the image capture mechanism such as desired to be output on monitor 720, then noise removal 
a sensor are stored first on camera 730 and downloaded later according to the various embodiments of the invention 
to be output, displayed, or manipulated. This allows the 30 should be employed prior to forming full color pixels 
camera 730 to capture the next object quickly without through color interpolation should be employed. The down- 
additional delay. One convenient raw image representation loaded and decompressed image is analyzed pixel-by-pixel 
is the 8-bit Bayer pattern CFA described above. and by applying the techniques discussed above, each pixel 

In one embodiment, an image is captured by camera 730 is transformed into a noise removed pixel. Preferably, each 

as a raw 8-bit Bayer pattern CFA and then compressed into 35 noise removed pixel is stored in memory 711 or disk 718 

some other data or encoded format. Noise removal accord- into an array so that the original (recovered) CFA is main- 

ing to one embodiment of the invention operates as follows. tained for the noise removal of other pixels. Once the noise 

First, the compressed image is downloaded from camera 730 removal is performed throughout the CFA, then the noise 

through I/O port 717 to a disk 718 or other device on the removed array may be used in the color interpolation 

computer system 710. The compressed image is decom- 40 process, and if desired, the original (recovered) CFA dis- 

pressed on computer system 710. Then, the computer system carded. Noise removed and then color interpolated rendered 

performs noise removal on the recovered CFA image prior image 750 will be rendered with better detail, sharper edges 

to any color interpolation by executing instructions for doing and less noise, given adequate display devices (monitor 720 

so as described in the various embodiments of the invention. and adapter 716) and will thus, more closely resemble the 

The methodology described in the various embodiments 45 original object/scene 740. The noise removed and color 

of the invention may be executed using a processor 712 such interpolated image may be stored onto disk 718, memory 

as the Pentium™ (a product of Intel Corporation) and a 711 and/or output directly to monitor 720 after being ren- 

memory 711, such as RAM, which is used to store/load dered through display adapter 716. 

instruction, addresses and result data. The application used The computer program code for noise removal may be 

to perform noise removal on the CFA image may be an 50 packaged onto a removable computer-readable medium such 

executable module compiled from source written in a lan- as floppy disk or CD-ROM and may accompany software 

guage such as C++. The instructions of that executable that performs the decompression of images downloaded 

module, which correspond with instructions that aid in from camera 730. As with any other software, it can be 

detecting edges and applying a first noise removal technique downloaded/distributed over a network on communications 

for edge pixels and a second noise removal technique for 55 system, or can be available in firmware. These downloaded 

non-edge pixels may be stored to a disk 718 or memory 711, images can be noise removed according to their edge 

and thus may be loaded through some computer-readable features so that the images will more accurately visually 

medium. It would be readily apparent to one of ordinary skill present the object/scene 740 to the user. Noise removal 

in the art of computer sciences to program a determining according to another embodiment of the invention may also 

machine to carry out the interpolation methodology 60 be implemented in hardware on the camera 730 itself, 

described in various embodiments of the invention. In an alternative embodiment of the invention, the image 

Computer system 710 has a system bus 713 which facili- processing circuit 732 can be adapted to include circuitry for 

tates information transfer to/from the processor 712 and performing noise removal on-camera, such that the stored 

memory 711 and also has a bridge 714 which couples system image in image memory unit 734 is a noise removed image, 

bus 713 to an I/O bus 715. I/O bus 715 connects various I/O 65 Thus, when downloaded to a computer system 710 or a 

devices such as a display adapter 716, disk 718 and an I/O rendering device, a first level of noise will be already 

port, such as a serial port. Many such combinations of I/O eliminated. In such an embodiment, the image may be noise 



11/04/2003, EAST version: 1.4.1 



US 6,229,578 Bl 



15 



16 



removed at any stage in the image processing. For instance, 
the noise removal technique(s) of the various embodiments 
of the invention, which are applicable directly in the CFA 
domain, may be employed after or before color resolution 

CGuipaiiuiLig. ili S ui,Li a liaiuvvai^ Cujl/GuiTljCiil, vviiClC SpvCu 5 

is desired, each noise removed pixel, as it is being 
determined, can be written back directly to the image 
memory unit 734 or other storage mechanism where the CFA 
is stored, so that the noise removal process is recursive. The 
methods described above for finding pixel gradients and 10 
determining edges are easily implemented in VLSI (Very 
Large Scale Integration) or other integrated circuitry. The 
noise removal techniques presented in various embodiments 
of the invention have the advantage of potential direct 
hardware implementation since unlike traditional noise 15 
removal techniques, they are designed to apply directly to 
the CFA domain. 

The exemplary embodiments described herein are pro- 
vided merely to illustrate the principles of the invention and 
should not be construed as limiting the scope of the inven- 20 
tion. Rather, the principles of the invention may be applied 
to a wide range of systems to achieve the advantages 
described herein and to achieve other advantages or to 
satisfy other objectives as well. 

What is claimed is: 25 

1. A method comprising: 

classifying single color pixels of a captured color image 
while in its Color Filter Array form as either edge pixels 
or non-edge pixels; 

applying a first noise removal technique to those pixels 30 
classified as non-edge pixels; and 

applying a second noise removal technique to those pixels 
classified as edge pixels, said noise removal techniques 
applied while said image is still in its Color Filter Array 
form, wherein classifying and applying are performed 35 
independently with respect to each color plane in said 
Color Filter Array such that said noise removal tech- 
niques are performed prior to color interpolation. 

2. A method according to claim 1 wherein classifying 
includes: 

defining a localization region, said localization region 
including a portion of said captured image; 

determining the gradient of every pixel within the local- 
ization region; and 45 

for each pixel, comparing its gradient value to a given 
threshold value, if the gradient value is greater than said 
threshold value, classifying the pixel as an edge pixel, 
and if not, classifying the pixel as a non-edge pixel. 

3. A method according claim 2 wherein said given thresh- 50 
old value is a localized threshold value determined by 
analyzing the gradient values for all pixels within the 
localization region. 

4. A method according to claim 2 wherein the steps of 
defining determining and classifying are repeated until all 55 
pixels of said captured image are classified. 

5. A method according to claim 1 wherein said first noise 
removal technique includes the application a multilevel 
median hybrid filter. 

6. A method according to claim 1 wherein said second eo 
noise removal technique includes a linear averaging of 
adjacent pixels belonging to the same edge feature. 

7. A method according to claim 2 wherein determining the 
gradient includes: 

applying a first mask to a neighborhood around each pixel 65 
whose gradient is being determined to attain a first 
direction differential quantity; 
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applying a second mask to said neighborhood around each 
pixel whose gradient is being determined to attain a 
second direction differential quantity, said second 
direction, wherein said second direction is perpendicu- 
lar to said first direction; 

determining the gradient to be the magnitude of a vector 
defined by vector addition of said first direction differ- 
ential quantity and said second direction differential 
quantity. 

8. A method according to claim 7 wherein said magnitude 
is obtained by: 

squaring said first direction differential; 
squaring said second direction differential; 
summing said squared first and second direction differ- 
entials; and 

extracting the square root of said sum, said magnitude set 
to equal said square root. 

9. A method according to claim 7 wherein said first mask 
is a Sobel mask having coefficients of -1 and +1 in variously 
said first direction. 

10. A method according to claim 7 wherein said first mask 
is a Sobel mask having coefficients of -1 and +1 variously 
in said second direction. 

11. A method according to claim 7 wherein said first mask 
is a Prewitt mask having coefficients of -1, -2, +1 and +2 
variously in said first direction. 

12. A method according to claim 7 wherein said first mask 
is a Prewitt mask having coefficients of -1, -2, +1 and +2 
variously in said second direction, 

wherein classifying and applying are performed indepen- 
dently with respect to each color plane in said CFA. 

13. A method according to claim 1 wherein said CFA 
includes four color planes, namely, a Red color plane, a first 
Green color plane, a second Green color plane and a Blue 
color plane. 

14. A method according to claim 1 wherein said CFA is 
arranged in a Bayer pattern. 

15. A method according to claim 6 wherein one edge 
feature is distinguished from a another edge feature by 
applying the steps of: 

determining the absolute value difference (AVD) between 
the pixel under consideration for noise removal and 
each pixel adjacent to said pixel under consideration for 
noise removal; 

averaging together all of said AVDs; 

comparing said average of AVDs with the AVD for each 
adjacent pixel, if said AVD for a given adjacent pixel is 
less than said average of AVDs, then including that 
adjacent pixel as belonging to the edge feature of said 
pixel under consideration for noise removal. 

16. A computer- readable medium having instructions 
stored thereon that when executed by a processor, cause said 
processor to perform: 

classifying single color pixels of a captured color image 
while in its color filter array form as either edge pixels 
or non-edge pixels; 

removing noise from a pixel classified as a non-edge pixel 
according to a first noise removal technique; and 

removing noise from a pixel classified as an edge pixel 
according to a second noise removal technique; 

wherein classification and noise removal are performed 
independently with respect to each color plane in said 
captured image such that said noise removal techniques 
are performed prior to color interpolation. 
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17. An imaging apparatus comprising: 

an image processing unit to receive raw color image data, 
said image processing unit configured to classify single 
color pixels in said raw image data; remove noise from 
uuu-cu^o piAcl* accuiuiug iu a nisi noise removal ~ 
technique and to remove noise from edge pixels 
according to a second noise removal technique; 
wherein classification and removal are performed inde- 
pendently with respect to each color plane in said raw 1Q 
image data such that said noise removal techniques are 
performed prior to color interpolation. 

18. An apparatus according to claim 17 wherein said raw 
image data is stored in a Bayer pattern color filter array. 

19. An apparatus according to claim 17 further compris- 15 
ing: 

an image memory unit coupled to said image processing 
circuit to receive and store said raw image data as 
pixels already subjected to said noise removal tech- 
niques. 20 

20. A system comprising: 

an image capture device configured to provide raw color 
image data in the Color Filter Array domain; and 
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a data processing unit coupled to said image capture 
device, said data processing unit configured to remove 
noise from said raw image data while still in the CFA 
domain, said data processing unit configured to classify 
single color pixels in said raw imape data: apply a first 
noise removal technique to pixels in said raw image 
data classified as non-edge pixels; and to apply a 
second noise removal technique to pixels in said raw 
image data classified as edge pixels; wherein classifi- 
cation and removal are performed independently with 
respect to each color plane in said raw image data such 
that said noise removal techniques are performed prior 
to color interpolation. 

21 . A system according to claim 20 further comprising a 
storage medium coupled to said data processing unit, said 
storage medium storing said noise removed image data. 

22, A system according to claim 20 further comprising a 
display device coupled to said data processing unit, said 
display device configured to render said noise removed raw 
image data together with interpolated color components to 
form a full color image upon output from said display 
device. 
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